Hidden Correlation Discovery: Towards the Automation of an Analysis System

IT Analytics company, Sumerian Ltd, has undertaken a 2-year project to automate and embed statistical techniques into its processes. Their current automated processes rely on correlation to highlight relationships between metric pairs. It is proposed that the nature of the data often obscures instances of high correlation, and that these may be revealed through the use of clustering analysis. Clustering analysis is a technique for dividing datasets into groups so that similar items are together and those displaying differences are separated. Like many data analysis methodologies, such techniques are often better suited to theoretical situations and are ill-equipped to perform well against the real-world situation of 'messy', massive-scale data. This project therefore aims to adapt simple clustering techniques to improve on the processes at Sumerian, by investigating different algorithms aimed at uncovering hidden correlation patterns within their datasets. Using the well-known k-means clustering algorithm as a starting point and baseline, we will attempt to find: highly-correlated subsets of any metric-pair without any categorical information; correlation in certain 'windows' of time within the data; and differences in the patterns of correlation between peak and other processing periods. Investigations into these developments will take in Hill Climbing and Genetic Algorithms. Overall, no single solution was found to fully meet the company's needs, but several avenues of future research have been uncovered, and recommendations are made on how to continue with the work. 3 Acknowledgements Huge thanks must go to my supervisor, Professor David Corne, for his support and guidance throughout both this and the wider KTP project, which would not have been remotely successful without his participation and encouragement. I would also like to thank Sumerian for allowing me to use my work with them as the basis for this dissertation, and particularly Chris Playford and George Theologou for rescuing me from total isolation during the work! I, Sarah Little, confirm that this work submitted for assessment is my own and is expressed in my words. Any uses made within it of the words of other authors in any form e.g. ideas, equations, figures, text, tables, programs are properly acknowledged at any point of their use. A list of the references employed is included.